Method and apparatus for generating 3d map of indoor space

ABSTRACT

According to an exemplary embodiment of the present disclosure, a three-dimensional map generating method of an indoor space, includes: obtaining at least one indoor space image which is an image for an indoor space; distinguishing a background area corresponding to a structure of the indoor space from a non-background area corresponding to objects located in the indoor space in the at least one indoor space image; generating at least one expanded indoor space image by expanding the background area to the non-background area in the at least one indoor space image; generating depth-image associated information based on at least one expanded indoor space image and geographic information including information of a depth value for the indoor space; and generating a three-dimensional map for the indoor space using the at least one expanded indoor space image, the geographic information, and the depth-image associated information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of Korean Patent Application No.10-2018-0040660 filed on Apr. 6, 2018 and Korean Patent Application No.10-2018-0116195 filed on Sep. 28, 2018 in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference.

BACKGROUND Field

The present disclosure relates to a method and an apparatus forgenerating a 3D map of an indoor space and more particularly, to a mapgenerating method and apparatus which three-dimensionally display anindoor space in a virtual space using information on an image and ashape of the indoor space.

Description of the Related Art

With regard to the indoor space modeling, traditionally, a method whichconverges a camera and a laser sensor to configure color point clouddata by mixing a geographic value obtained by a laser sensor and animage value obtained by a camera has been widely used and companies suchas FARO, RieGL, or LeiCA in the construction industry commercially applythe method.

Further, a three-dimensional space may be modeled by a method whichobtains color point cloud data using Kinect or a 3D sensor having asimilar operating principle and estimates a movement trajectory of asensor to accumulate the accumulated data on the moving trajectory.

However, the technology of the related art has the following problems.

First, various types of errors (a sensing error, a sensor positionestimation error, and a matching error between multiple sensors) relatedto the sensor measurement affect a three-dimensional map of an indoorspace to lower the immersion feeling of the user. For example, referringto FIG. 1, it is found that image information to be formed on a columnis formed on a bottom due to a small position estimation error of animage sensor so that it is difficult to create a realisticthree-dimensional map.

Second, it may be difficult to perfectly model a shape of a complexindoor structure in an arbitrary indoor environment so as to match withpixels of the image one to one. For example, referring to FIG. 2, it isunderstood that inconsistency between an image and a shape occurs andthus it is difficult to create a realistic three-dimensional map.Specifically, when there is a plurality of objects having very complexshapes such as chandelier, after an interior process, one-to-onematching of a shape and an image is a task that is actually impossiblewith current systems because it exceeds a measurement error level of thesensor itself.

Third, in order to completely emulate an arbitrary indoor environment,data on all areas of the indoor space such as under the desk or a narrowgap needs to be obtained. In order to obtain the data, a complex sensorsystem is necessary and data needs to be obtained by such a sensorsystem for a long time. Further, when the technology of the related artis performed, there is a limit in that if there is a moving person or amoving object, there is no sufficient time to measure the moving personor object by a sensor and the moving person or object moves separatelyfrom the movement of the sensor system so that geographic informationtherefor cannot be created. Therefore, when the existing technology isused, it is inconvenient because data needs to be obtained when there isno moving person or object or if there is a moving person or object, auser needs to wait until the moving person or object is out of a viewingangle of a sensor.

Therefore, there is a necessity for a new three-dimensional mapgenerating method and apparatus to solve the existing problems causedwhen the three-dimensional map is generated.

As a prior art, there is Korean Registered Patent No. 10-1835434(entitled “projected image generation method, device, and method formapping image pixels and depth values, published on Jan. 18, 2017).

SUMMARY

An object to be achieved by the present disclosure is to provide amethod and an apparatus for generating a three-dimensional map similarto an indoor space in a virtual space by utilizing information obtainedfrom an indoor space.

Technical problems to be solved by the present invention are not limitedto the above-mentioned technical problem(s), and other technicalproblem(s), which is (are) not mentioned above, can be clearlyunderstood by those skilled in the art from the following descriptions.

According to an aspect of the present disclosure, a three-dimensionalmap generating method of an indoor space includes: obtaining at leastone indoor space image which is an image for an indoor space;distinguishing a background area corresponding to a structure of theindoor space from a non-background area corresponding to objects locatedin the indoor space in at least one indoor space image; generating atleast one expanded indoor space image by expanding the background areato the non-background area in at least one indoor space image;generating a depth-image associated information based on at least oneexpanded indoor space image and geographic information includinginformation of a depth value for the indoor space; and generating athree-dimensional map for an indoor space using the at least oneexpanded indoor space image, the geographic information, and thedepth-image associated information.

Desirably, before the distinguishing of the non-background area, thethree-dimensional map generating method may further include: obtainingdepth value information which is information of a depth value for theindoor space; and generating indoor space outline information which isoutline information for the indoor space using the depth valueinformation and the geographic information may include the depth valueinformation and the indoor space outline information.

Desirably, after the generating of depth-image associated information,the three-dimensional map generating method may further include updatingat least one of the geographic information and the depth-imageassociated information using at least one expanded indoor space imageand features included in the geographic information, the generating of athree-dimensional map may use at least one of the updated geographicinformation and the depth-image associated information.

Desirably, between the generating of depth-image associated informationand the generating of a three-dimensional map for the indoor space, thethree-dimensional map generating method may further includere-distinguishing the background area from the non-background area inthe at least one indoor space image using the geographic information andthe depth-image associated information; and regenerating at least oneexpanded indoor space image by expanding the background area to thenon-background area in the at least one indoor space image and thegenerating of a three-dimensional map may use the regenerated at leastone expanded indoor space image.

Desirably, in the re-distinguishing of the background area from thenon-background area, when the background area and the non-backgroundarea are re-distinguished for one target image among at least one indoorspace image, the depth-image associated information and at least onesupplementary image excluding the target image from at least one indoorspace image are further used.

Desirably, the generating of a three-dimensional map for the indoorspace may include: selecting one expanded indoor space image associatedwith a reference pose corresponding to a predetermined referenceposition on a three-dimensional map among the at least one expandedindoor space image, based on the depth-image associated information todetermine the selected expanded indoor space image as a representativeimage; determining at least one supplementary image excepting therepresentative image among the at least one expanded indoor space image,using the depth-image associated information and the reference pose; andgenerating a three-dimensional map using the representative image, theat least one supplementary image, the geographic information, and thedepth-image associated information.

Desirably, the three-dimensional map generating method may furtherinclude: obtaining object information including information on at leastone object from an object DB including information on shapes and imagesof a plurality of objects; and adding at least one virtual objectcorresponding to the at least one object on the generatedthree-dimensional map, based on the object information.

Desirably, the object information may be information on an objectlocated in the indoor space.

Desirably, the three-dimensional map generating method may furtherinclude: selectively updating the at least one virtual object added tothe three-dimensional map when information on a new object is added tothe object DB or information on the existing virtual object is updated.

Desirably, the three-dimensional map generating method may furtherinclude: displaying a reflected image which is an image for an indoorspace which is reflected by the reflective virtual object with respectto the reference pose corresponding to a predetermined referenceposition on the three-dimensional map in the reflective virtual objectusing the at least one expanded indoor space image and the geographicinformation when a reflective virtual object which is a virtual objectincluding a material which reflects light is included in the at leastone virtual object.

Desirably, the three-dimensional map generating method may furtherinclude: estimating light source information including information on aposition and brightness of a light source located in the indoor spacefrom at least one indoor space image; and reflecting a lighting effectby the light source information to at least one expanded indoor spaceimage.

According to another aspect of the present disclosure, athree-dimensional map generating apparatus of an indoor space includes:an obtaining unit which obtains at least one indoor space image which isan image for an indoor space; a distinguishing unit which distinguishesa background area corresponding to a structure of the indoor space froma non-background area corresponding to objects located in the indoorspace in the at least one indoor space image; an expanding unit whichexpands the background area to the non-background area in the at leastone indoor space image to generate at least one expanded indoor spaceimage; an associating unit which generates a depth-image associatedinformation based on the at least one expanded indoor space image andgeographic information including information of a depth value for theindoor space; and a generating unit which generates a three-dimensionalmap for the indoor space using the at least one expanded indoor spaceimage, the geographic information, and the depth-image associatedinformation.

Desirably, the obtaining unit further obtains depth value informationwhich is information of a depth value for the indoor space and theapparatus may further include an outline information generating unitwhich generates indoor space outline information which is outlineinformation for the indoor space using the depth value information, andthe geographic information includes the depth value information and theindoor space outline information.

Desirably, the associating unit may update at least one of thegeographic information and the depth-image associated information usingthe at least one expanded indoor space image and features included inthe geographic information and the generating unit may use at least oneof the updated geographic information and the depth-image associatedinformation.

Desirably, the distinguishing unit may re-distiguish the background areafrom the non-background area in the at least one indoor space imageusing the geographic information and the depth-image associatedinformation, the expanding unit may regenerate at least one expandedindoor space image by expanding the background area to thenon-background area in the at least one indoor space image, and thegenerating unit may use the regenerated at least one expanded indoorspace image.

Desirably, when the distinguishing unit re-distinguishes the backgroundarea from the non-background area for one target image among the atleast one indoor space image, the distinguishing unit may further usethe depth-image associated information and at least one supplementaryimage excluding the target image from the at least one indoor spaceimage.

Desirably, the generating unit may select one expanded indoor spaceimage associated with a reference pose corresponding to a predeterminedreference position on a three-dimensional map among the at least oneexpanded indoor space image, based on the depth-image associatedinformation to determine the selected expanded indoor space image as arepresentative image, determine at least one supplementary imageexcepting the representative image, from the at least one expandedindoor space image, using the depth-image associated information and thereference pose, and generate a three-dimensional map using therepresentative image, the at least one supplementary image, thegeographic information, and the depth-image associated information.

Desirably, the three-dimensional map generating apparatus may furtherinclude: a DB connecting unit which obtains object information includinginformation on at least one object from an object DB includinginformation on shapes and images of a plurality of objects; and thegenerating unit may add at least one virtual object corresponding to theat least one object on the generated three-dimensional map, based on theobject information.

Desirably, the object information is information on an object located inthe indoor space.

Desirably, when information on a new object is added to the object DB orinformation on the existing virtual object is updated, the generatingunit may selectively update the at least one virtual object added to thethree-dimensional map.

Desirably, when a reflective virtual object which is a virtual objectincluding a material which reflects light is included in the at leastone virtual object, the generating unit may further display a reflectedimage which is an image for an indoor space which is reflected by thereflective virtual object with respect to the reference posecorresponding to a predetermined reference position on thethree-dimensional map in the reflective virtual object using the atleast one expanded indoor space image and the geographic information.

Desirably, the three-dimensional map generating apparatus may furtherinclude: a light source estimating unit which estimates light sourceinformation including information on a position and brightness of alight source located in the indoor space from the at least one indoorspace image; and a light source reflecting unit which reflects alighting effect by the light source information to the at least oneexpanded indoor space image.

According to the present disclosure, even though there are various typesof errors related to the sensor measurement, a 3D map even in anenvironment having various errors may be robustly and stably built byutilizing only a background part of the space which is relatively lessaffected by the error. Specifically, according to the present invention,robustness and stability are sought by positively utilizing a boundaryportion of the background, and more particularly, an inner boundary lineformed by a background portion with a non-background portion is enhancedto expand the background portion. Further, an edge of the image and anedge of a geography in the background are corrected to enhance mutualmatchability so that a 3D map may be robustly and stably built.

Further, according to the present disclosure, a methodology which, afterstably configuring a background, provides a non-background portion to befurnished on the background by utilizing information on images andshapes accumulated in a previously constructed object database (DB) isemployed. By doing this, it is possible to precisely represent an objectwhich is not only sensitive to errors but also does not have preciselyconstructed information from sensor information due to itscharacteristics utilizing the previously constructed object DB. Further,when a new object is registered or updated in the object DB with a timeinterval, more precise image and shape information are reflected toobjects on the previously generated map through the updated object DBand all the processes may be performed by a computer algorithm withoutan additional effort.

Further, according to the present disclosure, even though only partialinformation on the background portion is obtained, the background may beconfigured and even though only partial information on thenon-background portion is obtained, it is possible to robustly estimatea type of an object and a shape of a pose. Therefore, entire informationis deduced and expressed by obtaining only partial information in ashort time without obtaining all data at all times for a long time.Further, according to the present disclosure, even when there aredynamic objects such as a person which are desired not to be expressedon a final map while obtaining the data, the dynamic objects areconsidered as a non-background portion to be removed. Specifically, whena single geography-multiple image expressing method embodied in thepresent disclosure is utilized, even though it is difficult to deduce orexpand the background from specific image information due to severeconcealment such as a high partition, the background portion may beprecisely and effectively restored utilizing image information obtainedfrom another location.

Further, according to the present disclosure, when map data isconfigured by utilizing a single geography-multiple image expressingmethod, a lighting effect of the indoor is naturally changed inaccordance with the movement of the user to express a realistic virtualmap.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and other advantages of thepresent disclosure will be more clearly understood from the followingdetailed description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a view of an example in which image information of a column isprojected on a floor to cause a visual error;

FIG. 2 is a view of an example in which a visual error is incurred on anouter appearance of an object on a three-dimensional map due toinconsistency of an image and a shape;

FIG. 3 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space according to an exemplary embodiment of thepresent disclosure;

FIG. 4 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space according to another exemplary embodiment ofthe present disclosure;

FIG. 5 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space using a representative image and asupplementary image according to an exemplary embodiment of the presentdisclosure;

FIG. 6 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space according to still another exemplaryembodiment of the present disclosure;

FIG. 7 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space according to still another exemplaryembodiment of the present disclosure;

FIG. 8 is a block diagram illustrating a three-dimensional mapgenerating apparatus of an indoor space according to an exemplaryembodiment of the present disclosure;

FIG. 9 is a block diagram illustrating a three-dimensional mapgenerating apparatus of an indoor space according to another exemplaryembodiment of the present disclosure;

FIG. 10 is a block diagram illustrating a three-dimensional mapgenerating apparatus of an indoor space according to still anotherexemplary embodiment of the present disclosure;

FIG. 11 is a view of a wheel-type mobile robot, a backpack type system,and a hand-type system equipped with a camera, an inertial sensor, and adistance measuring sensor as an example of a sensor system for dataacquisition;

FIG. 12 is a view of a three-dimensional geographic informationgenerated using a Lidar sensor;

FIG. 13 is a view in which 360-degree panoramic image and a depth valueare associated with each other in the form of mesh;

FIGS. 14 and 15 are views illustrating an indoor space where no objectis located and an indoor space where various types of objects arelocated;

FIGS. 16 and 17 are views illustrating a background area and anon-background area of an indoor space, respectively;

FIG. 18 is a view illustrating a result obtained by expanding abackground area according to an exemplary embodiment of the presentdisclosure to a non-background area;

FIG. 19 is a view illustrating edges included in an expanded indoorspace image and indoor structure outline information according to anexemplary embodiment of the present disclosure;

FIG. 20 is a view illustrating an example in which areas on an image arepartitioned to have the same area in geographic information;

FIG. 21 is a view illustrating various exemplary embodiments ofgeographic information formed by point cloud data;

FIG. 22 is a view illustrating various exemplary embodiments ofgeographic information configured in the form of mesh; and

FIG. 23 is a view of an exemplary embodiment in which an indoor space isrepresented by a depth-image associated form with respect to a specificpose and an expanded indoor space image is utilized to represent anarbitrary field of view in a user viewer.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Those skilled in the art may make various modifications to the presentinvention and the present invention may have various embodimentsthereof, and thus specific embodiments will be illustrated in thedrawings and described in detail in the detailed description. However,this does not limit the present invention within specific exemplaryembodiments, and it should be understood that the present inventioncovers all the modifications, equivalents and replacements within thespirit and technical scope of the present invention. In the descriptionof respective drawings, similar reference numerals designate similarelements.

Terms such as first, second, A, or B may be used to describe variouscomponents but the components are not limited by the above terms. Theabove terms are used only to discriminate one component from the othercomponent. For example, without departing from the scope of the presentinvention, a first component may be referred to as a second component,and similarly, a second component may be referred to as a firstcomponent. A term of and/or includes a combination of a plurality ofrelated elements or any one of the plurality of related elements.

It should be understood that, when it is described that an element is“coupled” or “connected” to another element, the element may be directlycoupled or directly connected to the other element or coupled orconnected to the other element through a third element. In contrast,when it is described that an element is “directly coupled” or “directlyconnected” to another element, it should be understood that no elementis present therebetween.

Terms used in the present application are used only to describe aspecific exemplary embodiment, but are not intended to limit the presentinvention. A singular form may include a plural form if there is noclearly opposite meaning in the context. In the present invention, itshould be understood that terminology “include” or “have” indicates thata feature, a number, a step, an operation, a component, a part or thecombination those of described in the specification is present, but doesnot exclude a possibility of presence or addition of one or more otherfeatures, numbers, steps, operations, components, parts or combinations,in advance.

If it is not contrarily defined, all terms used herein includingtechnological or scientific terms have the same meaning as thosegenerally understood by a person with ordinary skill in the art. Termsdefined in generally used dictionary shall be construed that they havemeanings matching those in the context of a related art, and shall notbe construed in ideal or excessively formal meanings unless they areclearly defined in the present application.

Hereinafter, exemplary embodiments according to the present inventionwill be described in detail with reference to accompanying drawings.

In order to implement the present disclosure, data needs to be obtainedfrom an indoor space utilizing a sensor system including a sensor whichis capable of obtaining image information. Specifically, since theindoor space is much broader and larger than a viewing angle of thesensor, the data needs to be obtained while moving the sensor so as tosufficiently store the data of the indoor space. In this case, thesensor may use a single camera or a plurality of cameras to measure adepth and an inertial sensor or a laser sensor which is capable ofmeasuring a distance may be combined to be used.

In this case, since a sensor value is obtained while movement in anindoor space, a three-dimensional pose from which individual sensor datais obtained may be different from each other. Therefore, a pose valuefrom which individual sensor data is obtained needs to be preciselyestimated to create a precise three-dimensional map and thus athree-dimensional pose may be estimated by utilizing a technique such assimultaneous localization and mapping (SLAM). However, when the sensorsystem utilizes a mobile robot as illustrated in FIG. 11, data isobtained at a fixed height, so that the present disclosure may beimplemented only by estimating a state vector of a two-dimensional pose.That is, data with consistency of a sensor obtaining pose may beobtained.

In the meantime, a configuration of the sensor value may vary dependingon a type of a sensor. For example, when the sensor is configured onlyby a signal camera, the sensor value is configured only by a cameraimage and the image is utilized to extract a feature in the image. Arelative distance between the features is estimated using the singlecamera and an absolute distance between the features is estimated usingthe plurality of cameras. Specifically, in the case of the singlecamera, a depth of the pixel may be estimated by utilizing theaccumulated image information without extracting a feature and in thecase of the plurality of cameras, the depth of pixel may be estimated byimages of the plurality of cameras or accumulated information thereof.

Further, when information of an additional inertial sensor or a depthmeasurement sensor is utilized together, the sensor information may beprocessed in accordance with a unique characteristic of each sensor. Forexample, as illustrated in FIG. 11, when inertial sensor information canbe obtained, the information is utilized to improve the performance ofthe SLAM or the information is used as prediction information for animage obtaining point at the time of processing the image information tofacilitate the correction for the image obtaining point. Further, anacceleration value or an angular velocity of the inertial information isutilized to estimate an actual movement distance and correct a scale ofa depth value extracted from a single camera or a plurality of cameras.

In the same context, when it is possible to obtain sensor informationwhich enables depth measurement in the unit of ray, such as LiDAR orKinect, an obtained pose of the data is estimated through SLAM and depthmeasurement data is enumerated based on the estimated pose to moreprecisely estimate a three-dimensional geographic information asillustrated in FIG. 12.

In this case, as illustrated in FIG. 21, the geographic information mayinclude a point cloud type (top) formed by collecting points measured atevery ray, a point cloud type (middle) formed by collecting pointscorresponding to a background excluding objects from point clouds, or apoint cloud type (bottom) in which a background portion blocked byobjects is restored based on a geographic continuity of the backgroundportion.

Further, as illustrated in FIG. 22, the geographic information expandsto be represented in the form of mesh (upper left: outdoor view point,lower left: indoor view point) or meshes corresponding to a backgroundfrom which objects are excluded in the mesh are collected (upper right:outdoor view point, lower right: indoor view point).

In this case, a method of representing raw data of the three-dimensionalmap may vary depending on a configuration and an implementing method ofthe sensor. Among them, when a 360-degree camera is used, as illustratedin FIG. 13, the obtained image is represented as a panoramic image andan estimated depth value is associated with the panoramic image in theform of mesh, a coordinate value compared with the obtained image or aspecific referential point of the mesh is stored as a 3D or 2D poseform. FIG. 13 is an example of data of a specific pose and raw data ofthe three-dimensional map may be configured by a plurality of datadepending on an area of the indoor space.

In the meantime, referring to FIG. 11, a three-dimensional mapgenerating apparatus of an indoor space according to an exemplaryembodiment of the present disclosure receives data required to generatea three-dimensional map from a wheel type mobile robot to which a sensorsystem including a camera, an inertial sensor, and a Lidar sensor isattached through wired or wireless communication. Alternatively, thethree-dimensional map generating apparatus of an indoor space accordingto an exemplary embodiment of the present disclosure is directly mountedin the wheel type mobile robot attached with a sensor system to generatea three-dimensional map. Alternatively, the three-dimensional mapgenerating apparatus of an indoor space according to an exemplaryembodiment of the present disclosure may be a backpack type system, asmall scanner, or a smart phone including a camera and an inertialsensor or may be mounted in a system including a smart phone expandingdevice to generate a three-dimensional map.

However, the three-dimensional map generating apparatus of an indoorspace according to an exemplary embodiment of the present disclosure isnot limited to use the system illustrated in FIG. 11 but may receivedata required to generate a three-dimensional map from a deviceincluding various sensor systems or may be mounted in various types ofdevices to generate a three-dimensional map.

FIG. 3 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space according to an exemplary embodiment of thepresent disclosure.

In step S310, a three-dimensional map generating apparatus obtains atleast one indoor space image which is an image for an indoor space.

For example, the three-dimensional map generating apparatus may obtainat least one indoor space image which is an image for an indoor space,from an external device attached with a camera or a camera which isattached to the three-dimensional map generating apparatus.

In this case, the indoor space image may be an image obtained byphotographing the indoor space in various locations in the indoor space.

In step S320, the three-dimensional map generating apparatusdistinguishes a background area corresponding to a structure of theindoor space from a non-background area corresponding to objects locatedin the indoor space in at least one indoor space image.

In this case, the background area may refer to a portion correspondingto only the structure of the indoor space in the indoor space image.Further, the non-background area may refer to a portion which is desirednot to be represented in a final map, such as objects located in theindoor space or a moving person in the indoor space image.

For example, referring to FIGS. 14 to 17, FIG. 15 illustrates the indoorspace image in which various objects (desks, book shelves, or chairs)are located in the indoor space. In this case, as illustrated in FIG.16, a portion from which objects are removed (displayed with a whitecolor) is a background and as illustrated in FIG. 17, portionscorresponding to the objects may be a non-background. In this case,referring to FIG. 14, the background may refer to a structure of abuilding itself which forms the indoor space or may include a structure(for example, a door or a window) attached to the building in additionto the structure of the building itself.

In the meantime, the three-dimensional map generating apparatus maydistinguish elements which may be discontinuous due to concealment byanother element among components existing at least one indoor spaceimage but actually continuously configured in the configuration of theactual environment as the background area. Further, thethree-dimensional map generating apparatus may distinguish an areaexcluding an area distinguished as a background area, as anon-background area in at least one indoor space image. In this case,the three-dimensional map generating apparatus may distinguish thebackground area from the non-background area based on a pixel value ofan image which forms at least one indoor space image.

In this case, the background area may be a portion which is configuredfirst in time in an environment where a person lives, as illustrated inFIG. 14. That is, the background is a combination of components existingat the time of completing a wallpapering or painting task and then asillustrated in FIG. 15, non-background elements (for example, variousobjects) may be put on the background area. Further, the background areamay be a portion which is partially blocked by other elements so thatthe data is not complete but is deduced to have a similarity to aportion which is not blocked to be reconstructed from the portion whichis not blocked using hole filling or inpainting technique. Further, thebackground area may be a portion which may block other objects such as alarge signboard or an information desk in the building but may have thematchability of the image and the geography coinciding with each otherat the edge portion of the object in all the data or cause thematchability of the image and the geography to coincide with each otherthrough a separate matching process.

In step S330, the three-dimensional map generating apparatus generatesat least one expanded indoor space image by expanding the backgroundarea to the non-background area in at least one indoor space image.

In this case, the three-dimensional map generating apparatus expands thebackground area to the non-background area using information on abackground area of at least one indoor space image to generate anexpanded indoor space image.

For example, referring to FIG. 16, the three-dimensional map generatingapparatus may expand the background area to the non-background areausing information on the background area in the indoor space image. Thatis, the three-dimensional map generating apparatus may deduce andreinforce a portion corresponding to the non-background area usinginformation on an edge included in the background area to generate anexpanded indoor space image.

More specifically, when the edge included in the background area isdisconnected at a boundary line 1610 with the non-background area, thethree-dimensional map generating apparatus may generate an expandedindoor space image through deduction that an extension line of the edgeis continued to the non-background area beyond the boundary line 1610between the background area and the non-background area.

In this case, the three-dimensional map generating apparatus specifiesone or more indoor space images other than a specific indoor space imageas a background supplement image and an area corresponding to thenon-background area of the specific indoor space image may be reducedusing information on the background supplement image.

In the meantime, referring to FIG. 19, the three-dimensional mapgenerating apparatus may partially perform the expansion as needed. Inthis case, 10% of expansion is performed in FIG. 18A, 30% of expansionis performed in FIG. 18B, and the entire expansion is performed in FIG.18C. If a user wants to model only the structure of the indoor space inwhich no object is provided and furnish arbitrary virtual objects in thevirtual indoor space, 100% of expansion as illustrated in FIG. 18C isalso possible.

In step S340, the three-dimensional map generating apparatus generates adepth-image associated information based on at least one expanded indoorspace image and geographic information including information of a depthvalue for the indoor space.

In this case, the depth-image associated information may be informationthat matches a depth value of the indoor space corresponding to pixelsof at least one expanded indoor space image.

To this end, the three-dimensional map generating apparatus may generatedepth-image associated information further using an image obtaining poseand a depth obtaining pose including information on obtaining locationsand obtaining angles of at least one expanded indoor space image and thegeographic information in addition to at least one expanded indoor spaceimage and the geographic information.

More specifically, the three-dimensional map generating apparatus mayfind a coordinate relationship in which all or some of pixels of anindividual image correspond to the depth value included in thegeographic information through the image obtaining pose associated withat least one expanded indoor space image. Further, this is repeatedlyapplied to the at least one expanded indoor space image to generatedepth-image associated information.

In the meantime, when a resolution of the image sensor is higher than aresolution of the distance sensor, an unmapped pixel which does not havea corresponding depth value may be generated in the pixels of at leastone expanded indoor space image. In this case, the three-dimensional mapgenerating apparatus may map an estimated depth value to an unmappedpixel using interpolation.

According to another exemplary embodiment, the three-dimensional mapgenerating apparatus may update at least one of the geographicinformation and the depth-image associated information using featuresincluded in at least one expanded indoor space image and the geographicinformation.

For example, referring to FIG. 13, when all or some of the pixels of theindividual image are interlinked to a specific reference coordinate andthe depth-image associated information obtained by the interlinkedrelationship of the reference coordinate and the geographic informationis given to map the geographic information to the image pixel, thethree-dimensional map generating apparatus may perform a feature basedmatching process of updating at least one of the reference coordinateand the geographic information to match the feature included in theexpanded indoor space image and the geographic information in accordancewith a predetermined criterion.

In this case, the features in the feature based matching process arepresent in each of the expanded indoor space image and the geographicinformation and interlinked to each other and more specifically, may beconfigured by edge information, point information, segment information,line information, planar piece information, plane information, or acombination thereof.

For example, referring to FIG. 19, when the edge 1910 of at least oneexpanded indoor space image and an edge 1920 of the geographicinformation do not match each other, the reference coordinate may bemodified to increase the matchability between the edge 1910 of theexpanded indoor space image and the edge 1920 of the geographicinformation.

In another exemplary embodiment, the three-dimensional map generatingapparatus may increase the matchability between the edge 1910 of theexpanded indoor space image and the edge 1920 of the geographicinformation by correcting the geographic information during the featurebased matching process.

That is, when at least one of the reference coordinate and thegeographic information is updated through the feature based matchingprocess as described above, the three-dimensional map generatingapparatus may update the depth-image associated information asillustrated in FIG. 13, based on the updated information. Further, thethree-dimensional map generating apparatus may use at least one of theupdated depth-image associated information and the geographicinformation to generate a three-dimensional map.

In still another exemplary embodiment, the three-dimensional mapgenerating apparatus re-distinguishes the background area from thenon-background area using the geographic information and the depth-imageassociated information to regenerate at least one expanded indoor spaceimage.

That is, the three-dimensional map generating apparatus may moreprecisely re-distinguish the background area from the non-backgroundarea using the depth-image associated information. More specifically, asillustrated in FIG. 20, the three-dimensional map generating apparatusmaps a boundary line 1620 (that is, a feature) corresponding to the sameplane in the geographic information to an image through the depth-imageassociated information and divides the inside of the boundary line 1610into A, B, and C utilizing the mapping information, and expands an imageof the background area to the non-background area utilizing the dividedinformation.

In this case, the features in the geographic information are present ineach of the image and the geographic information and interlinked to eachother and more specifically, may be configured by edge information,point information, segment information, line information, planar pieceinformation, plane information, or a combination thereof.

Further, the three-dimensional map generating apparatus expands thebackground area which is re-distinguished in the indoor space image to anon-background area to regenerate at least one expanded indoor spaceimage. Further, the three-dimensional map generating apparatus may usethe regenerated at least one expanded indoor space image to generate athree-dimensional map.

In still another exemplary embodiment, when the three-dimensional mapgenerating apparatus re-distinguishes the background area from thenon-background area for one target image of at least one indoor spaceimage, the three-dimensional map generating apparatus may further usethe depth-image associated information and at least one supplementaryimage excluding the target image from at least one indoor space image.

That is, the three-dimensional map generating apparatus may specifyparts far from the reference coordinate among pixels in a specifictarget image using the depth-image associated information and expand thebackground area to the non-background area by simultaneously utilizingat least one supplementary image and the depth-image associatedinformation other than the specific image for the corresponding parts.Further, for an image which is not easy to estimate a background due toconcealment in the target image, at least one supplementary image havingmore excellent image information therefor is specified through thedepth-image associated information and the background area may beexpanded to the non-background area by utilizing the supplementaryimage.

Finally, in step S350, the three-dimensional map generating apparatusgenerates a three-dimensional map for an indoor space using the at leastone expanded indoor space image, the geographic information, and thedepth-image associated information.

In this case, the three-dimensional map generating apparatus maygenerate a more realistic three-dimensional map by mapping at least oneexpanded indoor space image to the geographic information based on thedepth-image associated information. The generated three-dimensional mapmay interlink with a user viewer which operates in a smart phone, a PC,and a tablet and the user may see the three-dimensional map for theindoor space through the user viewer.

Further, the three-dimensional map generating apparatus performs thefeature based matching process as described above to improve realismexperienced by the user.

In the meantime, when a field of view to be represented by a user viewerin the indoor space exceeds a field of view which is capable of beingrepresented by a single expanded indoor space image, thethree-dimensional map generating apparatus may represent the field ofview of the user by utilizing the depth-image associated information andone or more expanded indoor space image.

More specifically, as illustrated in FIG. 23, a reference position 2310of a field of view to be represented by the user viewer is differentfrom reference positions 2320, 2330, 2340, and 2350 of expanded indoorspace images 2321, 2331, 2341, and 2351 having depth-image associatedinformation, the field of view of the user viewer cannot be representedby the single expanded indoor space image and lost field of view may begenerated.

In this case, a plurality of expanded indoor space images required torepresent all fields of view of the reference position 2310 of the fieldof view of the user is specified by the depth-image associatedinformation and an expanded indoor space image associated with a pose inproximity to a reference pose of the user viewer is designated as arepresentative image and at least one expanded indoor space image otherthan the representative image is designated as a supplementary image torepresent all the fields of view of the user viewer simultaneouslyutilizing the representative image and at least one supplementary image.

In the meantime, a detailed method of generating a three-dimensional mapusing a representative image and at least one supplementary image by thethree-dimensional map generating apparatus will be described in detailwith reference to FIG. 5.

In another exemplary embodiment, the three-dimensional map generatingapparatus may generate geographic information using at least one indoorspace image.

For example, the three-dimensional map generating apparatus may extracta feature in an image using at least one indoor space image and mayestimate a relative distance between the features in the case of asingle camera and may estimate an absolute distance between the featuresin the case of a plurality of cameras. Further, in the case of thesingle camera, the three-dimensional map generating apparatus mayestimate a depth of the pixel utilizing the accumulated imageinformation without extracting a feature and in the case of theplurality of cameras, may estimate the depth of pixel by images of theplurality of cameras or accumulated information thereof.

By doing this, the three-dimensional map generating apparatus maygenerate geographic information using at least one indoor space image.

As described above, the three-dimensional map generating method of anindoor space according to an exemplary embodiment of the presentdisclosure utilizes only a background part of the space which isrelatively less affected by the error to robustly and stably build a 3Dmap even in an environment having various errors.

FIG. 4 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space according to another exemplary embodiment ofthe present disclosure.

In step S410, a three-dimensional map generating apparatus obtains atleast one indoor space image which is an image for an indoor space.

In step S420, the three-dimensional map generating apparatus obtainsdepth value information which is information on a depth value for anindoor space.

For example, the three-dimensional map generating apparatus may obtainthe depth value information for an indoor space, from an external devicemounted with a depth measurement sensor or a depth measurement sensorattached to the three-dimensional map generating apparatus.

In step S430, the three-dimensional map generating apparatus generatesindoor space outline information which is outline information for theindoor space using the depth value information.

Here, the indoor space outline information may be outline information ona structure of the indoor space where there is no object.

In the meantime, the three-dimensional map generating apparatus maygenerate indoor space outline information of a state in which objects inthe indoor space are removed, using an algorithm such as a convex cut.Here, the convex cut algorithm is an algorithm which extracts astructure of the indoor space in real time using depth value information(point cloud data) of the indoor space.

In step S440, the three-dimensional map generating apparatusdistinguishes a background area corresponding to a structure of theindoor space from a non-background area corresponding to objects locatedin the indoor space in at least one indoor space image.

In step S450, the three-dimensional map generating apparatus generatesat least one expanded indoor space image by expanding the backgroundarea to the non-background area in at least one indoor space image.

In step S460, the three-dimensional map generating apparatus generatesdepth-image associated information based on at least one expanded indoorspace image and geographic information including information of a depthvalue for the indoor space.

Finally, in step S470, the three-dimensional map generating apparatusgenerates a three-dimensional map for an indoor space using the at leastone expanded indoor space image, the geographic information, and thedepth-image associated information.

Here, the geographic information may include the depth value informationand the indoor space outline information.

FIG. 5 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space using a representative image and asupplementary image according to an exemplary embodiment of the presentdisclosure.

In step S510, the three-dimensional map generating apparatus selects oneexpanded indoor space image associated with a reference posecorresponding to a predetermined reference position on athree-dimensional map among at least one expanded indoor space image,based on the depth-image associated information and determines theselected expanded indoor space image as a representative image.

For example, the three-dimensional map generating apparatus determines apose of a user viewer among at least one expanded indoor space image asa reference pose and selects one expanded indoor space image associatedwith a pose closest to the reference pose to be determined as arepresentative image.

In step S520, the three-dimensional map generating apparatus determinesat least one supplementary image excepting the representative image,from at least one expanded indoor space image, using the depth-imageassociated information and the reference pose.

For example, the three-dimensional map generating apparatus mayadditionally determine at least one supplementary image close to thereference pose.

Finally, in step S530, the three-dimensional map generating apparatusgenerates a three-dimensional map using the representative image, the atleast one supplementary image, the geographic information, and thedepth-image associated information.

That is, the three-dimensional map generating apparatus may representall fields of view corresponding to the user viewer by simultaneouslyutilizing the representative image and the at least one supplementaryimage.

FIG. 6 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space according to still another exemplaryembodiment of the present disclosure.

In step S610, a three-dimensional map generating apparatus obtains atleast one indoor space image which is an image for an indoor space.

In step S620, the three-dimensional map generating apparatusdistinguishes a background area corresponding to a structure of theindoor space from a non-background area corresponding to objects locatedin the indoor space in at least one indoor space image.

In step S630, the three-dimensional map generating apparatus generatesat least one expanded indoor space image by expanding the backgroundarea to the non-background area in at least one indoor space image.

In step S640, the three-dimensional map generating apparatus generatesdepth-image associated information based on at least one expanded indoorspace image and geographic information including information of a depthvalue for the indoor space.

In step S650, the three-dimensional map generating apparatus generates athree-dimensional map for an indoor space using the at least oneexpanded indoor space image, the geographic information, and thedepth-image associated information.

In step S660, the three-dimensional map generating apparatus obtainsobject information including information on at least one object from anobject DB including information on shapes and images of a plurality ofobjects.

In this case, the object DB may include information on a type, a name, ashape, and an image for each of various types of objects. Further, thethree-dimensional map generating apparatus may reproduce athree-dimensional outer appearance of the object in a virtual space bycombining information on a shape of an arbitrary object and informationon an image. In the meantime, information on the shape of the object mayinclude information on a size of the object.

In the meantime, the three-dimensional map generating apparatus mayobtain object information on at least one object to be located on thegenerated three-dimensional map from the object DB. In this case, atleast one object to be located on the map may be an object which isactually located in the indoor space or an arbitrary object which is notlocated in the indoor space.

In another exemplary embodiment, the three-dimensional map generatingapparatus may identify an object located in the indoor space using atleast one expanded indoor space image.

To this end, the three-dimensional map generating apparatus may utilizean object DB including image and shape information of various objectsand utilize features information extracted from the information includedin the object DB or the image and shape information of the object toutilize for segmentation or classification of the object.

When it is determined that the identified object does not exist in theobject DB or exists in another form, the three-dimensional mapgenerating apparatus may request the user to add or update shape andimage information on a new object or a modified object to the object DB.

Alternatively, the three-dimensional map generating apparatus may searchobjects having the most similar size and shape from the object DBinstead of finding the perfectly same object as the identified object.This is because since the background portion of the expanded indoorspace image is larger than the background portion of the indoor spaceimage, even though an object which is larger than the object existing inthe non-background area is selected from the object DB to be disposed,visual incompatibility is not caused. In contrast, even though an objectwhich is larger than the object existing in the non-background area isselected to be disposed, the background behind the object is built sothat there is no data deficiency or awkward portions between thebackground and the object.

Finally, in step S670, the three-dimensional map generating apparatusadds at least one virtual object corresponding to at least one object tothe generated three-dimensional map based on the object information.

That is, the three-dimensional map generating apparatus locates at leastone virtual object corresponding to at least one object included in theobject information on the three-dimensional map to complete athree-dimensional map including at least one virtual object.

In another exemplary embodiment, when information on a new object isadded to the object DB or information on the existing virtual object isupdated, the three-dimensional map generating apparatus may selectivelyupdate at least one virtual object added to the three-dimensional map.

That is, when a new object is registered or updated in the object DBwith a time interval, the three-dimensional map generating apparatus mayreflect the registered or updated new information to a virtual objectlocated on the previously generated three-dimensional map. Further, thethree-dimensional map generating apparatus repeats the process by acomputer algorithm to consistently update the information of the objectlocated on the three-dimensional map without additional effort.

Further, when information on a new object is added to the object DB orinformation of the existing virtual object is updated, thethree-dimensional map generating apparatus may selectively update theinformation of the virtual object only when the information of thevirtual object located on the three-dimensional map needs to be updated.

For example, when the three-dimensional map generating apparatus triesto add an object A on the three-dimensional map, if there is noinformation on the object A in the object DB, the three-dimensional mapgenerating apparatus may add a virtual object corresponding to an objectB similar to the object A on the three-dimensional map. However, wheninformation on the object A is newly added to the object DB thereafter,the three-dimensional map generating apparatus replaces the virtualobject corresponding to the object B with the virtual objectcorresponding to the object A.

Similarly, when information on the object A is updated in the object DBafter adding the object A on the three-dimensional map, thethree-dimensional map generating apparatus may update the virtual objectcorresponding to the object A.

In another exemplary embodiment, when a reflective virtual object whichis a virtual object including a material which reflects light isincluded in at least one virtual object, the three-dimensional mapgenerating apparatus may further display a reflected image for theindoor space reflected by the reflective virtual object to thereflective virtual object.

Here, the material which is included in the reflective virtual object toreflect light may include a mirror, a glass, and a metal, but is notlimited thereto. Further, the information on the shape and the image ofthe plurality of objects included in the object DB may further includeinformation on transmittance and an area of the reflective material aswell as whether the plurality of objects includes a material whichreflects light.

When the reflective virtual object is included in at least one virtualobject, the three-dimensional map generating apparatus may display areflected image reflected through the reflective virtual object withrespect to the reference pose corresponding to the reference position ofthe three-dimensional map on a surface of the reflective virtual object.

For example, the three-dimensional map generating apparatus may furtherdisplay an image for the indoor space reflected through the reflectivevirtual object when the user who views the three-dimensional map seesthe reflective virtual object with respect to the reference pose, on thesurface of the reflective virtual object. This is because the image forthe indoor space is not reflected to the reflective virtual objectincluded in the object DB.

To this end, the three-dimensional map generating apparatus mayconfigure new depth-image associated information corresponding to anarea of the reflective material included in the reflective virtualobject using at least one expanded indoor space image and the geographicinformation. Alternatively, the three-dimensional map generatingapparatus may extract new depth-image associated informationcorresponding to an area of the reflective material included in thereflective virtual object from the previously generated depth-imageassociated information.

The three-dimensional map generating apparatus may generate a reflectedimage to be represented on the reflective material when the user seesthe reflective virtual object at the reference pose by consideringtransmittance of the reflective material based on the new depth-imageassociated information. Further, the three-dimensional map generatingapparatus may represent the reflected image in the area of thereflective material included in the reflective virtual object.

More specifically, when the reflective material is a mirror, thethree-dimensional map generating apparatus may generate an imageprojected on a mirror when the mirror is viewed in an arbitrary positionin the indoor space to display the image on the reflective virtualobject. Further, when the reflective material is a translucent glass,the three-dimensional map generating apparatus may overlap a reflectedimage proportional to the transmittance of the translucent glass and thetransmitted image when the translucent glass is viewed in an arbitraryposition in the indoor space to display the overlapping image on thereflective virtual object.

As described above, when the three-dimensional map generating apparatusupdates the reflected image to the reflective virtual object in realtime while continuously changing a point of view of the user, an effectthat the image of the indoor space projected on the reflective materialsuch as a mirror changes in the same manner as the actual change isassigned, so that a more realistic three-dimensional map may begenerated.

FIG. 7 is a flowchart illustrating a three-dimensional map generatingmethod of an indoor space according to still another exemplaryembodiment of the present disclosure.

In step S710, a three-dimensional map generating apparatus obtains atleast one indoor space image which is an image for an indoor space.

In step S720, the three-dimensional map generating apparatusdistinguishes a background area corresponding to a structure of theindoor space from a non-background area corresponding to objects locatedin the indoor space in at least one indoor space image.

In step S730, the three-dimensional map generating apparatus generatesat least one expanded indoor space image by expanding the backgroundarea to the non-background area in at least one indoor space image.

In step S740, the three-dimensional map generating apparatus generates adepth-image associated information based on at least one expanded indoorspace image and geographic information including information of a depthvalue for the indoor space.

In step S750, the three-dimensional map generating apparatus generates athree-dimensional map for an indoor space using the at least oneexpanded indoor space image, the geographic information, and thedepth-image associated information.

In step S760, the three-dimensional map generating apparatus estimateslight source information including information on a position andbrightness of a light source located in the indoor space from at leastone indoor space image.

That is, the three-dimensional map generating apparatus may estimate aposition and brightness of a light source using information of a pixelvalue included in the image, from at least one indoor space image.

For example, the three-dimensional map generating apparatus may estimatea position corresponding to a pixel value within a predetermined rangeas a position of the light source from at least one indoor space imageand estimate a brightness of the light source in accordance with thepixel value of the estimated location.

Finally, in step S770, the three-dimensional map generating apparatusreflects a lighting effect by the light source information to at leastone expanded indoor space image.

For example, during the process of generating an expanded indoor spaceimage by expanding the background area to the non-background area by thethree-dimensional map generating apparatus, the lighting effect for thenon-background area may disappear. In this case, the three-dimensionalmap generating apparatus reflects the lighting effect by the lightsource effect to the expanded indoor space image to generate a morerealistic three-dimensional map.

Further, when at least one virtual object is located on thethree-dimensional map, the three-dimensional map generating apparatusfurther adds a shadow for at least one virtual object in accordance withthe position and the brightness of the light source included in thelight source information to improve realism of the three-dimensionalmap.

In another exemplary embodiment, when the three-dimensional mapgenerating apparatus reflects the lighting effect to thethree-dimensional map, the lighting effect is reflected through apre-procedure before the user watches the map without reflecting thelighting effect in a real-time environment when the user views the mapso that a load in accordance with a real-time lighting effectcomputation may be reduced.

As a result, when the three-dimensional map generating apparatusconfigures the three-dimensional map by utilizing one indoor structureoutline information and at least one expanded indoor space image, areal-time load proportional to the number of at least one expandedindoor space image may be reduced.

FIG. 8 is a block diagram illustrating a three-dimensional mapgenerating apparatus of an indoor space according to an exemplaryembodiment of the present disclosure.

Referring to FIG. 8, a three-dimensional map generating apparatus 800 ofan indoor space according to an exemplary embodiment of the presentdisclosure includes an obtaining unit 810, a distinguishing unit 820, anexpanding unit 830, an associating unit 840, and a generating unit 850.Further, the three-dimensional map generating apparatus 800 may furtherinclude an outline information generating unit (not illustrated) as anoption.

The obtaining unit 810 obtains at least one indoor space image which isan image for an indoor space.

The distinguishing unit 820 distinguishes a background areacorresponding to a structure of the indoor space from a non-backgroundarea corresponding to objects located in the indoor space in at leastone indoor space image.

The expanding unit 830 generates at least one expanded indoor spaceimage by expanding the background area to the non-background area in atleast one indoor space image.

The associating unit 840 generates depth-image associated informationbased on at least one expanded indoor space image and geographicinformation including information on a depth value for the indoor space.

The generating unit 850 generates a three-dimensional map for the indoorspace using the at least one expanded indoor space image, the geographicinformation, and the depth-image associated information.

Finally, the outline information generating unit (not illustrated)generates indoor space outline information which is outline informationfor the indoor space using the depth value information.

In another exemplary embodiment, the obtaining unit 810 further obtainsdepth value information which is information of a depth value for theindoor space and the geographic information may include the depth valueinformation and the indoor space outline information generated by theoutline information generating unit (not illustrated).

In another exemplary embodiment, the associating unit 840 updates atleast one of the geographic information and the depth-image associatedinformation using at least one expanded indoor space image and featuresincluded in the geographic information and the generating unit 850 usesat least one of the updated geographic information and depth-imageassociated information.

In another exemplary embodiment, the distinguishing unit 820re-distinguishes the background area and the non-background area usingthe geographic information and the depth-image associated information inat least one indoor space image, the expanding unit 830 expands thebackground area in at least one indoor space image to the non-backgroundarea to regenerate at least one expanded indoor space image, and thegenerating unit 850 uses the regenerated at least one expanded indoorspace image.

In still another exemplary embodiment, when the distinguishing unit 820re-distinguishes the background area from the non-background area forone target image of at least one indoor space image, the distinguishingunit may further use the depth-image associated information and at leastone supplementary image excluding the target image from at least oneindoor space image.

In another exemplary embodiment, the generating unit 850 selects oneexpanded indoor space image associated with a reference posecorresponding to a predetermined reference position on athree-dimensional map among at least one expanded indoor space image,based on the depth-image associated information and determines theselected expanded indoor space image as a representative image,determines at least one supplementary image excepting the representativeimage, from at least one expanded indoor space image, using thedepth-image associated information and the reference pose, and generatesa three-dimensional map using the representative image, at least onesupplementary image, the geographic information, and the depth-imageassociated information.

FIG. 9 is a block diagram illustrating a three-dimensional mapgenerating apparatus of an indoor space according to another exemplaryembodiment of the present disclosure.

Referring to FIG. 9, a three-dimensional map generating apparatus 800 ofan indoor space according to an exemplary embodiment of the presentdisclosure includes an obtaining unit 810, a distinguishing unit 820, anexpanding unit 830, an associating unit 840, a generating unit 850, anda DB connecting unit 860.

The obtaining unit 810 obtains at least one indoor space image which isan image for an indoor space.

The distinguishing unit 820 distinguishes a background areacorresponding to a structure of the indoor space from a non-backgroundarea corresponding to objects located in the indoor space in at leastone indoor space image.

The expanding unit 830 generates at least one expanded indoor spaceimage by expanding the background area to the non-background area in atleast one indoor space image.

The associating unit 840 generates depth-image associated informationbased on at least one expanded indoor space image and geographicinformation including information on a depth value for the indoor space.

The generating unit 850 generates a three-dimensional map for the indoorspace using the at least one expanded indoor space image, the geographicinformation, and the depth-image associated information.

Finally, the DB connecting unit 860 obtains object information includinginformation on at least one object from an object DB includinginformation on shapes and images of a plurality of objects.

In another exemplary embodiment, the generating unit 850 may add atleast one virtual object corresponding to at least one object on thegenerated three-dimensional map, based on the object information.

In another exemplary embodiment, the object information may beinformation on an object located in the indoor space.

In another exemplary embodiment, when information on a new object isadded to the object DB or information on the existing virtual object isupdated, the generating unit 850 may selectively update at least onevirtual object added to the three-dimensional map. In another exemplaryembodiment, when a reflective virtual object which is a virtual objectincluding a material which reflects light is included in at least onevirtual object, the generating unit 850 may further display a reflectedimage which is an image for an indoor space which is reflected by thereflective virtual object with respect to the reference posecorresponding to a predetermined reference position on thethree-dimensional map in the reflective virtual object using at leastone expanded indoor space image and the geographic information.

FIG. 10 is a block diagram illustrating a three-dimensional mapgenerating apparatus of an indoor space according to still anotherexemplary embodiment of the present disclosure.

Referring to FIG. 10, a three-dimensional map generating apparatus 800of an indoor space according to an exemplary embodiment of the presentdisclosure includes an obtaining unit 810, a distinguishing unit 820, anexpanding unit 830, an associating unit 840, a generating unit 850, alight source estimating unit 870, and a light source reflecting unit880.

The obtaining unit 810 obtains at least one indoor space image which isan image for an indoor space.

The distinguishing unit 820 distinguishes a background areacorresponding to a structure of the indoor space from a non-backgroundarea corresponding to objects located in the indoor space in at leastone indoor space image.

The expanding unit 830 generates at least one expanded indoor spaceimage by expanding the background area to the non-background area in atleast one indoor space image.

The associating unit 840 generates depth-image associated informationbased on at least one expanded indoor space image and geographicinformation including information on a depth value for the indoor space.

The generating unit 850 generates a three-dimensional map for the indoorspace using the at least one expanded indoor space image, the geographicinformation, and the depth-image associated information.

The light source estimating unit 870 estimates light source informationincluding information on a position and a brightness for a light sourcelocated in the indoor space from at least one indoor space image.

Finally, the light source reflecting unit 880 reflects a lighting effectby the light source information to at least one expanded indoor spaceimage.

The above-described exemplary embodiments of the present invention maybe created by a computer-executable program and implemented in a generaluse digital computer which operates the program using acomputer-readable recording medium.

The computer-readable recording medium includes a magnetic storagemedium (for example, a ROM, a floppy disk, and a hard disk) and anoptical reading medium (for example, CD-ROM and a DVD).

For now, the present invention has been described with reference to theexemplary embodiments. It is understood to those skilled in the art thatthe present invention may be implemented as a modified form withoutdeparting from an essential characteristic of the present invention.Therefore, the disclosed exemplary embodiments may be considered by wayof illustration rather than limitation. The scope of the presentinvention is presented not in the above description but in the claimsand it may be interpreted that all differences within an equivalentrange thereto may be included in the present invention.

What is claimed is:
 1. A three-dimensional map generating method of an indoor space, comprising: obtaining at least one indoor space image which is an image for an indoor space; distinguishing a background area corresponding to a structure of the indoor space from a non-background area corresponding to objects located in the indoor space in the at least one indoor space image; generating at least one expanded indoor space image by expanding the background area to the non-background area in the at least one indoor space image; generating a depth-image associated information based on the at least one expanded indoor space image and geographic information including information of a depth value for the indoor space; and generating a three-dimensional map for the indoor space using the at least one expanded indoor space image, the geographic information, and the depth-image associated information.
 2. The three-dimensional map generating method according to claim 1, further comprising: before the distinguishing of the non-background area, obtaining depth value information which is information of a depth value for the indoor space; and generating indoor space outline information which is outline information for the indoor space using the depth value information, wherein the geographic information includes the depth value information and the indoor space outline information.
 3. The three-dimensional map generating method according to claim 1, further comprising: after the generating of depth-image associated information, updating at least one of the geographic information and the depth-image associated information using the at least one expanded indoor space image and features included in the geographic information, wherein the generating of a three-dimensional map uses at least one of the updated geographic information and depth-image associated information.
 4. The three-dimensional map generating method according to claim 1, further comprising: between the generating of depth-image associated information and the generating of a three-dimensional map for the indoor space, re-distinguishing the background area from the non-background area in the at least one indoor space image using the geographic information and the depth-image associated information; and regenerating at least one expanded indoor space image by expanding the background area to the non-background area in the at least one indoor space image; wherein the generating of a three-dimensional map uses the regenerated at least one expanded indoor space image.
 5. The three-dimensional map generating method according to claim 4, wherein in the re-distinguishing of the background area from the non-background area, when the background area and the non-background area are re-distinguished for one target image among the at least one indoor space image, the depth-image associated information and at least one supplementary image excluding the target image from the at least one indoor space image are further used.
 6. The three-dimensional map generating method according to claim 1, wherein the generating of a three-dimensional map for the indoor space includes: selecting one expanded indoor space image associated with a reference pose corresponding to a predetermined reference position on the three-dimensional map among the at least one expanded indoor space image, based on the depth-image associated information to determine the selected expanded indoor space image as a representative image; determining at least one supplementary image excepting the representative image among the at least one expanded indoor space image, using the depth-image associated information and the reference pose; and generating a three-dimensional map using the representative image, the at least one supplementary image, the geographic information, and the depth-image associated information.
 7. The three-dimensional map generating method according to claim 1, further comprising: obtaining object information including information on at least one object from an object DB including information on shapes and images of a plurality of objects; and adding at least one virtual object corresponding to the at least one object on the generated three-dimensional map, based on the object information.
 8. The three-dimensional map generating method according to claim 7, wherein the object information is information on an object located in the indoor space.
 9. The three-dimensional map generating method according to claim 7, further comprising: selectively updating the at least one virtual object added to the three-dimensional map when information on a new object is added to the object DB or information on the existing virtual object is updated.
 10. The three-dimensional map generating method according to claim 7, further comprising: displaying a reflected image which is an image for the indoor space which is reflected by the reflective virtual object with respect to the reference pose corresponding to a predetermined reference position on the three-dimensional map in the reflective virtual object using the at least one expanded indoor space image and the geographic information when a reflective virtual object which is a virtual object including a material which reflects light is included in the at least one virtual object.
 11. The three-dimensional map generating method according to claim 1, further comprising: estimating light source information including information on a position and brightness of a light source located in the indoor space from the at least one indoor space image; and reflecting a lighting effect by the light source information to the at least one expanded indoor space image.
 12. A three-dimensional map generating apparatus of an indoor space, comprising: an obtaining unit which obtains at least one indoor space image which is an image for an indoor space; a distinguishing unit which distinguishes a background area corresponding to a structure of the indoor space from a non-background area corresponding to objects located in the indoor space in the at least one indoor space image; an expanding unit which expands the background area to the non-background area in the at least one indoor space image to generate at least one expanded indoor space image; an associating unit which generates depth-image associated information based on the at least one expanded indoor space image and geographic information including information of a depth value for the indoor space; and a generating unit which generates a three-dimensional map for the indoor space using the at least one expanded indoor space image, the geographic information, and the depth-image associated information.
 13. The three-dimensional map generating apparatus according to claim 12, wherein the obtaining unit further obtains depth value information which is information of a depth value for the indoor space, an outline information generating unit which generates indoor space outline information which is outline information for the indoor space using the depth value information is further provided, and the geographic information includes the depth value information and the indoor space outline information.
 14. The three-dimensional map generating apparatus according to claim 12, wherein the associating unit updates at least one of the geographic information and the depth-image associated information using the at least one expanded indoor space image and features included in the geographic information and the generating unit uses at least one of the updated geographic information and the depth-image associated information.
 15. The three-dimensional map generating apparatus according to claim 12, wherein the distinguishing unit re-distinguishes the background area from the non-background area in the at least one indoor space image using the geographic information and the depth-image associated information, the expanding unit regenerates at least one expanded indoor space image by expanding the background area to the non-background area in the at least one indoor space image, and the generating unit uses the regenerated at least one expanded indoor space image.
 16. The three-dimensional map generating apparatus according to claim 15, wherein when the distinguishing unit re-distinguishes the background area from the non-background area for one target image among the at least one indoor space image, the distinguishing unit further uses the depth-image associated information and at least one supplementary image excluding the target image from the at least one indoor space image.
 17. The three-dimensional map generating apparatus according to claim 12, wherein the generating unit selects one expanded indoor space image associated with a reference pose corresponding to a predetermined reference position on the three-dimensional map among the at least one expanded indoor space image, based on the depth-image associated information to determine the selected expanded indoor space image as a representative image; determines at least one supplementary image excepting the representative image, from the at least one expanded indoor space image, using the depth-image associated information and the reference pose; and generates a three-dimensional map using the representative image, the at least one supplementary image, the geographic information, and the depth-image associated information.
 18. The three-dimensional map generating apparatus according to claim 12, further comprising: a DB connecting unit which obtains object information including information on at least one object from an object DB including information on shapes and images of a plurality of objects; and wherein the generating unit adds at least one virtual object corresponding to the at least one object on the generated three-dimensional map, based on the object information.
 19. The three-dimensional map generating apparatus according to claim 18, wherein the object information is information on an object located in the indoor space.
 20. The three-dimensional map generating apparatus according to claim 18, wherein when information on a new object is added to the object DB or information on the existing virtual object is updated, the generating unit selectively updates the at least one virtual object added to the three-dimensional map.
 21. The three-dimensional map generating apparatus according to claim 18, wherein when a reflective virtual object which is a virtual object including a material which reflects light is included in the at least one virtual object, the generating unit further displays a reflected image which is an image for the indoor space which is reflected by the reflective virtual object with respect to the reference pose corresponding to a predetermined reference position on the three-dimensional map in the reflective virtual object using the at least one expanded indoor space image and the geographic information.
 22. The three-dimensional map generating apparatus according to claim 12, further comprising: a light source estimating unit which estimates light source information including information on a position and brightness of a light source located in the indoor space from the at least one indoor space image; and a light source reflecting unit which reflects a lighting effect by the light source information to the at least one expanded indoor space image. 